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B.  Executive  Summary 


The  objective  of  this  research  effort  has  been  to  develop  and  validate  technical 
approaches  for  evaluating  the  effects  of  Knowledge  Based  Software  Assistant  (KBSA)  process 
concepts  and  technology  on  software  development  effort  and  schedule,  and  to  use  the  approaches 
to  perform  comparative  evaluations  of  KBSA  and  other  sources  of  software  technology. 

The  research  approach  has  involved  four  tasks. 

Task  1.  Characterize  KBSA  and  other  sources  of  software  technology  in  the  context  of 
recent  and  emerging  software  trends.  We  provide  a  sununary  of  KBSA  technology, 
concentrating  on  the  KBSA  Advanced  Development  Model  developed  by  Andersen  Consulting. 
We  also  summarize  two  other  comparative  sources  of  software  technology;  the  commercial 
marketplace  and  the  DARPA/AFRL  Evolutionary  Design  of  Complex  Software  (EDCS) 
program.  To  our  knowledge,  this  is  the  first  software  technology  payoff  analysis  done  in 
comparison  to  concurrent  advances  in  commercial  software  technology. 

Task  2.  Develop  models  and  an  evaluation  framework  for  assessing  the  effects  of  KBSA 
and  other  sources  of  software  technology  on  software  development  effort  and  schedule.  Our 
recently  developed  and  calibrated  COCOMO  II  model  provides  an  approach  for  evaluation  based 
on  the  effects  of  alternative  software  technologies  on  the  model's  effort-driver  parameters.  The 
model's  calibration  to  over  100  1990's  software  projects  also  provides  a  1990's  baseline  fi’om 
which  to  evaluate  the  technologies'  effects. 

For  assessing  schedule  effects,  we  have  developed  another  model,  CORADMO,  for 
evaluating  the  effects  of  rapid  application  development  (RAD).  Our  evaluation  framework  also 
includes  a  domain  focus:  DoD  warfighting  systems;  and  a  particular  evaluation  example:  a 
representative  embedded,  high-assurance,  real-time  (EHART)  missile  software  project.  We  have 
also  developed  a  spreadsheet  version  of  the  evaluation  model.  This  enables  technology  decision 
makers  to  perform  tradeoff  and  sensitivity  analyses  of  alternative  software  technology 
investment  strategies. 

Task  3.  Use  the  models  to  evaluate  KBSA,  EDCS,  and  commercial  technology  with 
respect  to  the  baseline.  The  primary  result  is  shovm  in  Figure  1,  which  compares  the 
technologies'  relative  effects  on  development  effort,  using  relatively  conservative  assumptions. 
It  shows  that  commercial  technology  is  likely  to  reduce  development  effort  of  the  EHART  1998 
baseline  project  by  a  factor  of  2.5  in  8  years  (2006)  and  another  factor  of  3  in  15  years  (2013). 
Relative  to  commercial  technology,  a  fully-supported  mix  of  KBSA  and  EDCS  technologies 
could  reduce  development  effort  by  another  factor  of  3  in  8  years  and  another  factor  of  6  in  15 
years.  These  assessment  results  are  consistent  with  such  analyses  of  commercial  software 
technology  impact  as  [Bernstein,  1997]. 
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Figure  1.  Effects  of  Commercial/Basic  DoD  (CD),  KBSA  (K),  and  EDCS/KBSA  (EK) 

Technology  on  Effort 
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Task  4.  Formulate  study  conclusions  and  recommendations. 

The  six  primary  conclusions  are: 

1.  The  KBSA  Advanced  Development  Model  was  not  sufficiently  robust  and  scalable  to 
provide  appreciable  benefits  to  critical  DoD  software  projects. 

2.  However,  the  KBSA  concepts,  if  otherwise  realized,  have  strong  potential  for  reducing 
costs  and  schedules  for  DoD-critical  warfighting  software  applications.  The  key  to  realizing  this 
potential  is  to  develop  the  KBSA  technology  in  the  context  of  DoD  warfighting  domain 
knowledge. 

3.  The  KBSA  Decision  Support  area  provides  an  attractive  new  direction  for  Air  Force 
and  DoD  software  technology  investments. 

4.  EDCS  technology  investments  and  payoffs  have  both  commonalities  and 
complementarities  with  respect  to  KBSA  technology  investments  and  payoffs. 

5.  The  Software  Technology  Impact  Analysis  spreadsheet  model  provides  a  useful  tool 
for  assessing  alternative  software  technology  assessment  strategies. 

6.  There  is  no  software  technology  silver  bullet  for  DoD  warfighting  systems,  including 
reliance  on  commercial  technology. 

The  two  primary  recommendations  are: 

1.  The  Air  Force  and  DoD  should  continue  a  strong  level  of  investment  in  software 
technology,  particularly  for  warfighting  domains. 

2.  The  software  technology  investments  should  be  part  of  an  integrated  DoD  software 
management/process/technology  strategy  to  ensure  DoD  software  supremacy,  particularly  in 
warfighting  domains. 

The  conclusions  and  recommendations  are  not  particularly  surprising.  They  are 
quantitative  counterparts  of  the  qualitative  conclusions  and  recommendations  about  the  need  for 
DoD  investments  in  software  technology  in  recent  Air  Force  Scientific  Advisory  Board  studies 
[SAB,  1995;  SAB,  1997]  and  recent  DoD-level  studies  [NAE,  1995;  Fox  et  1995;  NRC, 
1997].  What  is  surprising  is  that  in  response  to  this  consistent  set  of  signals,  DoD  has  been 
reducing  its  level  of  investments  in  software  technology. 
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C.  Objectives  of  the  Effort 


The  objectives  of  this  research  effort  have  been  to  develop  and  validate  technical 
approaches  for  evaluating  the  effects  of  Kno^vledge  Based  Software  Assistant  (KBSA)  process 
concepts  and  technology  on  software  development  effort  and  schedule,  and  to  use  the  approaches 
to  perform  comparative  evaluations  of  KBSA  and  other  sources  of  software  technology. 

These  objectives  relate  to  the  strong  Air  Force  needs  for  improved  software  technology, 
particularly  in  areas  not  well  served  by  commercial  software  technology.  These  needs  are 
expressed  in  such  Air  Force  Scientific  Advisory  Board  studies  as  New  World  Vistas  [SAB, 
1995]  and  Air  Expeditionary  Forces  [SAB,  1997].  Similar  needs  have  been  expressed  at  the 
DoD  level  by  recent  reviews  of  DoD/DARPA  software  technology  programs  [NAE,  1995;  Fox  et 
al.,  1995],  and  the  National  Research  Council  “Ada  and  Beyond”  report  [NRC,  1997]. 
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D.  Approach 

The  research  approach  has  involved  four  tasks: 

Task  1 .  Characterize  KBSA  and  other  sources  of  software  technology  in  the  context  of 
recent  and  emerging  software  trends.  Section  E.l  summarizes  KBSA  technology,  concentrating 
on  the  KBSA  Advanced  Development  Model  developed  by  Andersen  Consulting.  It  also 
summarizes  two  other  comparative  sources  of  software  technology:  the  commercial  marketplace 
and  the  DARPA/AFRL  Evolutionary  Design  of  Complex  Software  (EDCS)  program.  Details  of 
the  KBSA  ADM  evaluation  are  in  Section  M. 

Task  2.  Develop  models  and  an  evaluation  framework  for  assessing  the  effects  of  KBSA 
and  other  sources  of  software  technology  on  software  development  effort  and  schedule.  Our 
recently  developed  and  calibrated  COCOMO II  model  provides  an  approach  for  evaluation  based 
on  the  effects  of  alternative  software  technologies  on  the  model's  effort-driver  parameters.  The 
model's  calibration  to  over  100  1990's  software  projects  also  provides  a  1990's  baseline  firom 
which  to  evaluate  the  technologies'  effects. 

Section  E.2  summarizes  COCOMO  II  and  an  additionally-developed  model,  CORADMO, 
for  evaluating  the  effects  of  rapid  application  development  (RAD)  strategies  on  software 
development  schedules.  It  also  summarizes  the  evaluation  domain  focus  —  DoD-critical 
embedded,  high-assurance,  real-time  (EHART)  projects  -  and  the  evaluation  timefi*ame:  current 
1998  baseline  efforts  and  schedules  on  a  representative  DoD  EHART  project  starting  in  1998, 
and  counterpart  project  efforts  and  schedules  if  the  project  were  started  in  2006  or  2013  with  the 
benefit  of  commercial  technology  or  well-supported  KBSA  and  EDCS  technology. 

Section  F  provides  a  summary  of  a  spreadsheet-based  Software  Technology  Impact  Analysis 
model,  which  can  be  used  to  evaluate  other  software  technologies  or  to  perform  sensitivity 
analyses  on  the  evaluations  performed  in  this  study.  Details  of  the  models  are  provided  in 
Section  J,  K,  and  O. 

Task  3.  Use  the  models  to  evaluate  KBSA,  EDCS,  and  commercial  technology  with  respect 
to  the  baseline.  Section  E.3  presents  example  worksheets  showing  the  technology  assessments  in 
terms  of  their  COCOMO  II  and  CORADMO  effort  and  schedule  driver  ratings,  with  rationales 
for  each.  The  primary  benefits  of  well-supported  KBSA  technology  come  from  domain-KB- 
based  applications  generation  and  KB  software  decision  aids,  particularly  in  the  EHART  domain 
and  other  DoD  warfighting  domains,  which  are  less  supported  by  commercial  technology.  The 
primary  benefits  of  well-supported  EDCS  technology  come  from  domain  architectures,  general 
architecture  and  collaboration  technology,  and  high-assurance  software  technology.  The 
complete  set  of  worksheets  and  spreadsheet  calculations  are  in  Section  L. 

Task  4.  Formulate  study  conclusions  and  recommendations. 

These  are  provided  in  Section  E.4,  and  were  summarized  in  the  Executive  Summary 
(Section  B). 
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E.  Discussion  of  Tasks 


E.l  KBSA  and  Other  Technologies 
E.1.1  The  1983  KBSA  Vision 

The  "Report  on  a  Knowledge  Based  Software  Assistant"  [Green  et  al.,  1983]  provided  a 
vision  for  achieving  major  improvements  in  software  development  and  maintenance.  It  involved 
the  following  primary  technical  advances; 

a.  Automatic  generation  of  desired  software  systems  from  formal  specifications. 

b.  Life  cycle  maintenance  and  evolution  of  software  via  modification  of  specifications  and  re¬ 
derivation  of  the  resulting  software,  rather  than  modifying  the  source  code  itself. 

c.  A  life  cycle  process  model  organized  around  development  of  specifications,  iteration  of 
automatic  program  generation  directives  to  achieve  desired  target  software  performance,  and 
subsequently  iterating  the  specifications  to  accommodate  software  changes. 

d.  Knowledge  based  "activity  coordination"  expertise  built  into  software  development  and 
management  tools,  to  ensure  effective  software  acquisition  and  evolution  management. 

E.l. 2  Developments  Since  1983 

E.  1 .2. 1  Evolution  of  Conventional  Technology  Toward  KBSA  Vision 

Since  1983,  conventional  software  development  technology  has  been  undergoing 
significant  changes,  reorienting  it  more  toward  aspects  of  the  KBSA  vision.  Large,  powerful 
COTS  software  components  (e.g.,  Microsoft  Foundation  Classes)  now  form  the  basis  for  many 
domain-specific  applications  generators,  such  as  spreadsheets,  business  fourth  generation 
languages  (4GL's),  and  graphic  user  interface  (GUI)  builders.  The  process  model  for  using  these 
packages  largely  follows  the  KBSA  process  model  (except  generally  for  the  lack  of  a 
performance  tuning  cycle),  as  it  is  much  more  natural  and  cost-effective  to  modify  applications 
at  the  4GL  level  than  at  the  target-code  level. 

Concurrently,  large  software  organizations  such  as  AT&T,  HP,  DoD,  and  its  contractors 
have  been  making  significant  investments  in  software  product  line  management  based  on  domain 
specific  software  architecture  (DSSA)-based  reusable  components.  The  DSSA's  are  based  on  a 
domain  engineering  process,  which  is  a  mix  of  systems  engineering  and  knowledge  engineering 
focused  on  the  applications  domain  of  interest. 

However,  significant  limitations  remain  with  respect  to  the  evolving  conventional 
technology.  Combinations  of  large  COTS  components  are  extremely  hard  to  integrate,  often 
leading  to  factors  of  4-5  in  budget  and  schedule  overruns  [Garlan  et  al.,  1995].  Various 
commercial  and  government  interface  standardization  initiatives  have  been  formed  to  address 
this  problem  (e.g.,  OMG's  CORBA,  DoD’s  Defense  Information  Infrastructure-Common 
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Operating  Environment  (DII-COE)).  Wrapper  and  glue-code  technology  has  been  developed  to 
address  the  problem  also,  but  the  problem  remains  significant. 

E.  1.2.2  Evolution  of  KBS  A  Technology 

The  AFRL/Information  Directorate  KBSA  program  began  with  a  number  of  exploratory 
research  efforts  addressing  the  various  facets  of  a  KBSA  capability.  These  facets  included 
prototype  KBSA  tools  supporting  software  project  management,  requirements  acquisition  and 
analysis,  formal  specification  development,  transformation  of  specifications  into  code,  and 
performance  optimization.  Some  additional  research  efforts  focused  on  facets  of  an 
infrastructure  capability  such  as  tool  integration,  activity  coordination,  and  communication 
coordination  [White,  1991]. 

Concurrently  with  the  later  stages  of  the  facets  research,  Andersen  Consulting  developed 
an  exploratory  Concept  Demonstration  version  of  an  overall  KBSA  system.  It  demonstrated  the 
integration  of  portions  of  the  requirements  acquisition,  code  development,  and  project 
management  facets  in  an  Air  Traffic  Control  domain  [De  Beilis  et  al.,  1992]. 

As  the  program  has  proceeded  toward  its  ultimate  objectives,  it  has  spawned  a  number  of 
productivity-enhancing  spinoffs  such  as  the  Refine-based  software  reengineering  and  testing 
tools,  and  domain-specific  applications  generators  covering  portions  of  avionics,  transportation 
planning,  and  military  message  processing  applications.  These  have  had  significant  productivity 
benefits  for  portions  of  a  process  or  product,  but  await  further  capabilities  to  be  able  to  address  a 
full  life  cycle  process  or  a  full  system  product. 

The  KBSA  activity-coordination  area  began  with  the  work  on  automating  configuration 
management  and  producing  software  project  management  decision  aids.  A  number  of  additional 
exploratory  software  decision  support  aids  have  been  produced  in  such  areas  as  risk  assessment, 
process  model  selection,  review  preparation,  and  schedule-slip  assessment,  e.g.,  [Bose  et  al., 
1995].  More  recent  AFRL-sponsored  work  has  been  for  decision  support  aids  for  DoD 
warfighting  domains  such  as  for  security  and  survivability. 

A  major  effort  to  integrate  KBSA  technology  has  been  the  KBSA  Advanced 
Development  Model  described  in  the  next  section. 

E.1.3  The  KBSA  Advanced  Development  Model 

The  KBSA  Advanced  Development  Model  developed  as  part  of  the  AFRL/Information 
Directorate  Laboratory's  Knowledge-Based  Software  Assistant  effort  is  aimed  at  improving 
software  development  productivity  and  software  quality.  The  fundamental  approach  in 
achieving  the  above  goal  is  providing  automated  support  that  mediates,  automates  and 
documents  all  activities  throughout  the  development  lifecycle  for  both  individual  developers  and 
teams  of  developers.  The  challenge  is  building  such  computer  based  assistants  as  elaborated  in 
the  KBSA  program  vision  [Green  et  al.,  1983]. 

The  key  concepts  to  meeting  the  above  challenge  are  based  on  the  understanding  that 
software  development  is  a  knowledge  intensive  activity.  Creating  large  software  based  systems 
requires  knowledge  of  the  domain  (typically  multi-disciplinary  in  nature),  knowledge  of  the 
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process  context,  knowledge  of  existing  components  and  hardware,  and  personnel  resources.  The 
KBSA  approach  is  then  to  provide  means  for  capture  and  effective  use  of  this  knowledge  with 
the  goal  that  such  use  of  this  knowledge  by  the  stakeholders  will  lead  to  timely  production  of 
high-quality  software. 

Given  the  above  understanding,  the  key  idea  in  the  KBSA  approach  is  exploiting  artificial 
intelligence  (AI)  concepts  and  representations  to  capture  and  use  knowledge.  The  different  types 
of  product  specific  knowledge  are  user  requirements,  system  specifications,  code,  test  scenarios 
and  documentation.  Process  specific  knowledge  corresponds  to  the  software  development  plans, 
resources  and  status  of  the  project.  Some  major  problems  here  are:  a)  integrated  usage  of  the 
knowledge  [Selfridge,  1992];  b)  managing  change;  and  c)  managing  complexity.  Significant 
progress  has  been  made  in  addressing  the  above  problems  individually  [Johnson  et  al.,  1991,  Mi 
et  al.,  1990,  Smith,  1991].  The  ADM  objective  was  to  build  on  such  advances  to  provide  an 
integrated  set  of  concepts  and  tools  that  address  the  problems  within  a  single  framework. 

The  goals  of  our  analysis  were  restricted  to  evaluating  the  ADM  concepts  and  support 
features.  In  particular  we  focused  on:  a)  understanding  the  ADM  conceptual  model,  b)  analyzing 
ADM  facilitated  softweire  development  processes,  c)  analyzing  ADM  in  the  context  of  specific 
application  development  case  study,  and  d)  mapping  ADM  contributions  into  the  parameters  of  a 
cost  benefit  evaluation  of  the  ADM  support  features.  This  report  documents  our  findings  with 
respect  to  the  above  goals.  Here  is  a  summary  of  the  findings: 

1.  The  ADM  environment  framework  has  good  technical  concepts  (model-view 
controller  architecture;  process-driven  environment;  persistent  object  base). 

2.  These  concepts  have  several  critical  issues  regarding  the  feasibility  of  their  use  on 
large,  complex  projects  (scalability;  overconstraining  people-collaboration  processes). 

3.  The  ADM  was  not  fully-enough  implemented  to  resolve  these  issues. 

4.  Much  of  the  ADM  has  been  overtaken  by  commercial  technology  (e.g..  Rational 
Rose). 

5.  The  ADM  has  some  good  concepts  such  as  the  use  of  critics  for  software  project 
decision  assistance  or  conceptual  debugging. 

6.  For  the  foreseeable  future,  KBSA  technology  will  have  more  impact  on  software 
project  costs  and  schedules  if  pursued  in  terms  of  specialized  tool  enhancements  of 
commercial  environments  (e.g.,  domain-specific  application  generators;  critics  or 
software  project  decision  aids),  rather  than  in  terms  of  alternative  environments. 

E.1.4  Related  Technology 

Most  technology  impact  analyses  simply  compare  the  downstream  effect  of  applying 
their  technology  to  the  results  achievable  with  current  practice.  This  is  generally  unsatisfactory, 
because  it  leaves  the  reader  with  no  way  to  assess  the  gains  via  the  technology  being  analyzed  to 
the  gains  which  would  have  accrued  via  commercial  technology  evolution,  non-technology  (e.g., 
process  and  management)  gains,  or  via  alternative  technologies. 
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With  our  COCOMO-oriented  baseline  of  current  practice  and  technology  assessment 
approach,  we  can  do  better  than  this.  In  addition  to  assessing  the  downstream  effect  of  KBSA 
technology,  we  also  assess  the  counterpart  downstream  effects  of  commercial  software 
technology,  DoD  non-technology  initiatives  (e.g.,  integrated  product  teams,  process  maturity, 
existing-asset  reuse  initiatives),  and  the  major  alternative  DoD  software  technology  initiative,  the 
DARPA/AFRL  Evolutionary  Design  of  Complex  Software  (EDCS)  program. 

Commercial  technology  can  reduce  software  costs  and  schedules  in  a  number  of  ways 
captured  by  COCOMO  cost  drivers.  Moore’s  Law  increases  in  hardware  speed  and  memory 
capacity  can  reduce  execution  time  and  storage  constraints.  Commercial  off-the-shelf  (COTS) 
software,  class  libraries,  and  open  architecture  frameworks  can  reduce  costs  and  schedules  via 
increased  reuse.  CASE  tool  improvements  reduce  software  specification,  development,  test,  and 
documentation  costs.  Commercial  network-based  collaboration  tools  can  reduce  both  costs  and 
schedules. 

The  EDCS  program  has  five  technology  clusters  focused  on  two  primary  themes.  One  is 
improved  support  for  incremental  software  evolution  (contrary  to  popular  perception,  evolution 
technology  begins  paying  off  not  only  after  delivery  but  after  Ae  first  specifications  are 
formulated).  The  second  is  technology  development  in  DoD  mission  contexts,  such  as  aircraft, 
missiles,  satellite  systems,  and  sensor  systems.  Its  five  technology  clusters  are  described  in  the 
recent  EDCS  Demo  Days  98  document  [Salasin  -  La  Monica,  1998]  as  follows: 

Rationale  Capture  and  Software  Understanding:  Rationale  is  captured  in  a  knowledge 
base  and  represented  in  a  manner  that  permits  explanation,  exploration,  and  management. 
Constraint  maintenance  techniques  can  be  used  to  monitor  rationale,  such  as  design  decisions,  so 
that  future  changes  to  the  system  do  not  violate  previous  requirements. 

Architecture  and  Generation:  Architecture  activities  combine  descriptions  of  component 
functionality  with  specifications  of  how  a  component  interacts,  coordinates,  cooperates,  and 
communicates  with  other  components.  The  goal  is  to  recognize  the  fundamental  qualities 
imparted  to  systems  by  these  various  interconnection  strategies,  verify  that  implementations 
achieve  these  qualities,  and  encourage  informed  choices.  Executable  code  can  be  generated  or 
composed  in  the  context  of  the  specified  architecture. 

High  Assurance  and  Real-Time:  Design,  development,  deployment,  integration, 
interoperation,  and  evolution  of  systems  that  provide  evidence  supporting  high  confidence  that 
all  critical  system  requirements  Avill  be  met.  Two  approaches  are  being  explored.  The  first 
approach  combines  testing  and  analysis  of  executable  code.  The  second  emphasizes  providing 
run-time  assurances  that  the  system  will  conform  to  desired  properties. 

Design  Management:  Addresses  a  scaleable  information  substrate  that  is  aimed  at  the 
retention,  organization,  and  exploitation  of  all  useful  software  information.  It  will:  provide 
shared  views  of  the  information  infrastructure;  support  information  conceptualization, 
representation,  and  manipulation  of  multi-media  software  artifacts;  enable  effective  collaboration 
in  a  distributed  setting;  provide  consistency  management  of  all  software-pertinent  artifacts;  and, 
support  processes  for  the  controlled  evolution  of  the  substrate. 
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Dynamic  Languages:  The  capabilities  of  dynamic  languages  to  support  incremental 
changes,  rapid  prototyping,  and  optimized  software  performance,  are  combined  with  techniques 
to  increase  reliability  and  efficiency,  and  facilitate  interoperability  with  more  conventional  static 
languages.  Dynamic  languages  also  promote  implementation  decisions  that  are  made  at  the 
latest  possible  stages  of  system  development  to  avoid  costly  rework. 

E.2  KBSA  Evaluation  Framework  and  Models 

E.2.1  Overall  Evaluation  Framework 

We  have  used  the  COCOMO  II  cost  and  schedule  estimation  model  [Boehm  et  al.,  1995] 
as  the  primary  model  used  in  performing  the  KBSA  impact  evaluation.  The  rationale  for  using 
COCOMO  II  involves  four  primary  factors: 

1 .  It  was  designed  as  a  revision  of  the  original  Constructive  Cost  Model  (COCOMO) 
[Boehm,  1981]  to  better  address  current  and  future  software  life  cycle  practices. 

2.  Its  internals  are  in  the  public  domain,  and  it  is  available  either  as  a  free  basic  tool  or 
in  several  more  powerful  commercial  versions. 

3.  It  is  well-calibrated  to  recent  projects.  The  COCOMO  11.1998  calibration  produces 
estimates  within  30%  of  project  actuals  71%  of  the  time  for  its  sample  of  161 
projects,  and  76%  of  the  time  when  its  coefficient  is  calibrated  to  individual 
organizations  [Chulani  et  al.,  1998]. 

4.  Its  project  database  provides  a  baseline  for  the  KBSA  evaluation.  Over  100  of  the 
projects  in  the  database  were  completed  in  the  1990’s.  Their  average  cost  driver 
ratings  provide  a  current-practices  baseline  from  which  we  can  assess  the  impact  on 
the  cost  drivers  of  KBSA  and  other  technologies.  We  can  then  use  the  model  to 
derive  the  resulting  cost  and  schedule  estimates  associated  with  these  resulting  cost 
drivers. 

We  have  found  that  the  COCOMO  II  schedule  estimation  model  is  good  for  traditional 
software  projects,  but  not  for  Applications  Composition  or  other  forms  of  Rapid  Application 
Development  (RAD).  COCOMO  II,  along  with  COCOMO  81  and  most  other  software  cost- 
schedule  estimation  models,  uses  a  variant  of  the  classic  cube-root  model  for  schedule 
estimation: 


M  =  3 

where  M  is  the  development  schedule  in  months  and  PM  is  the  development  effort  in 
person-months. 

Recently,  many  organizations,  including  DoD  mission-critical  system  organizations,  have 
found  that  it  is  more  important  to  minimize  software  development  or  modification  schedule 
(cycle  time)  than  to  minimize  cost.  They  have  also  found  a  number  of  techniques,  such  as  rapid 
prototyping,  development  process  reengineering  and  critical  path  streamlining,  collaboration 
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technology,  architecture-based  parallel  development,  and  prepositioning  of  assets,  which  can 
make  schedules  faster  than  the  cube  root  model. 

bi  concert  with  its  Affiliates,  USC-CSE  has  developed  a  RAD  Opportunity  Tree 
providing  a  structural  approach  to  such  methods  [Boehm-Chulani-Egyed,  1997].  We  have 
recently  completed  a  COCOMO  n  extension  called  CORADMO  for  estimating  the  schedule 
effects  of  various  mixes  of  RAD  strategies  (see  sections  E.2.2  and  K  for  further  details). 

Figure  2  provides  an  overview  of  how  we  use  COCOMO  n  and  CORADMO  to  perform 
the  software  technology  impact  assessments  for  the  representative  DoD  embedded,  high- 
assurance,  real-time  (EHART)  missile  software  project. 


Resulting  1998 
EHART  cost, 
schedule  estimates 


2006, 2013  EHART-CD 
cost,  schedule  estimates 

2006, 2013  EHART- 
KBSA  cost  schedule 

2006, 2013  EHART- 
EDCS  cost  schedule 
estimates 

2006, 2013  EHART- 
KBSA/EDCS  cost, 
schedule  estimates 


Figure  2.  Overview  of  Impact  Assessment  Approach 
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E.2.2  Evaluation  Approach 


Figure  3  provides  more  specifics  on  how  the  evaluation  is  conducted.  It  shows  the  ranges 
of  the  effort  multipliers  and  scale  factors  in  the  1998  calibration  of  the  COCOMO  II  model.  A 
diamond  on  each  of  the  ranges  shows  the  average  multiplier  for  the  106  1990’s  projects  in  the 
COCOMO  II  database. 


♦  Average  Multiplier  for  1990’s  projects 

Figure  3.  COCOMO  11.1998  Productivity  Ranges  and  Current  Practice 

For  example,  the  range  of  multipliers  for  the  TOOL  (Use  of  Software  Tools)  cost  driver 
runs  fi'om  a  multiplier  of  1.17  for  a  Very  Low  project  use  of  software  tools  (17%  less  efficient 
than  a  project  using  a  Nominal  level  of  tools),  to  a  multiplier  of  0.78  for  a  Very  High  project  use 
of  software  tools  (22%  more  efficient).  Detailed  rating  scales  for  TOOL  and  the  other  variables 
are  in  Section  J. 

In  the  COCOMO  family  of  models,  software  development  or  modification  effort  in 
person  months  (PM)  is  calculated  as: 

PM  =  ATI  (cost  driver  multipliers)  (Effective  Size)®^^*^®*® 

Cost  is  calculated  from  PM  via  an  average  cost/PM  factor.  Effective  size  is  converted 
into  equivalent  thousands  of  lines  of  source  code  (KSLOC),  including  effects  of  reuse,  breakage, 
ftmction  points  and  language  level. 

The  average  TOOL  multiplier  for  the  106  1990’s  projects  in  the  COCOMO  II  database  is 
1.01.  For  TOOL  and  each  of  the  other  COCOMO  II  cost  drivers  and  scale  factors,  we  have 
provided  a  worksheet  in  Section  L  providing  our  assessments  of  the  effects  of 
commercial/normal  DoD,  KBSA,  and  EDCS  improvements  on  the  project’s  TOOL  rating  and 
multiplier  for  the  years  2006  and  2013. 
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For  example,  our  assessment  was  that  commercial  technology  would  decrease  the 
project’s  TOOL  multiplier  from  1 .01  to  0.94  by  the  year  2013  (about  1/3  of  the  way  to  the  lowest 
multiplier  of  0.78).  We  assessed  that  KBSA  technology  would  lower  the  multiplier  farther,  to 
0.90;  and  that  the  combination  of  KBSA  and  EDCS  technology  would  lower  the  project’s  TOOL 
multiplier  to  0.86,  about  2/3  of  the  way  to  the  lowest  multiplier.  The  rationale  for  these 
reductions  was  that  KBSA  and  EDCS  teclmology  would  be  providing  stronger  tools  for  EHART 
applications  than  mainstream  commercial  technology  would  provide. 

Once  all  the  cost  driver  multipliers  and  scale  factors  (and  the  reductions  in  effective  size 
due  to  reuse,  very  high  level  languages,  and  reduced  breakage)  were  assessed,  we  could  enter 
them  into  COCOMO  II  to  generate  estimates  of  the  required  effort  and  schedule  for  the  various 
combinations  of  technologies  and  assessment  dates.  A  similar  approach  was  used  to  adjust  the 
estimates  for  RAD  improvement  via  CORADMO. 

E.2.3  Overview  of  COSSEMO  and  CORADMO  Models 

E.2.3.1  Introduction 


The  COCOMO  RAD  MODEL  (CORADMO)  is  currently  implemented  in  two  parts:  a 
front  end  staged  schedule  and  effort  model,  COCOMO  Stage  Schedule  and  Effort  MODEL 
(COSSEMO),  and  a  back  end  RAD  model 


E.2.3.2  COCOMO  Stage  Schedule  and  Effort  MODEL  (COSSEMO) 


The  COSSEMO  model  is  based  on  the  lifecycle  anchoring  concepts  discussed  by 
[Boehm,  1996].  The  anchor  points  are  defined  as  Life  Cycle  Objectives  (LCO),  Life  Cycle 
Architecture  (LCA),  and  Initial  Operational  Capability  (IOC).  Figure  4,  adapted  from  Rational 
Corporation  [Rational,  1998]  shows  the  stages  around  the  anchor  points. 


Activity  ievels  vs.  time 


LCO  LCA 


IOC 


rganization 

long 

ontent 


Stages 

Process  Components 

Requirements  Capture 
Analysis  &  Design 

Implementation 
Test  _ _ 

Supporting  Components 

Management 
Environment 
Depioyment 


Iterations 


Figure  4.  COCOMO  Stages  (Rational  Phases)  and  Anehor  Point  Milestones 
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COCOMO  II,  COSSEMO  &  CORADMO  use  the  word  "stage"  so  it  is  not  confused  with  the 
classic  waterfall  phases  which  correspond  to  the  activities  shown  in  Figure  4:  Requirements 
Capture,  Analysis  &  Design,  Implementation,  and  Test. 

COCOMO's  effort  and  schedule  estimates  are  focused  on  Elaboration  and  Construction 
(the  Stages  between  LCO  and  IOC).  Inception  corresponds  to  the  COCOMO's  "Requirements" 
activity,  which  is  actually  an  additional  (fixed  percentage)  effort,  above  and  beyond  the  effort 
calculated  by  COCOMO. 

Another  important  difference  of  COSSEMO/CORADMO’s  schedule  estimation  from 
COCOMO  II's  simple  schedule  estimation  is  the  use  of  a  more  complex  calculation  for  the  low 
effort  situations.  The  initial  COCOMO  II  baseline  schedule  equation  roughly  follows  the  classic 
“three  times  the  cube  root  of  the  effort”  model.  For  low-effort  situations,  for  example  PM  = 
twenty  seven  (27)  person  months,  this  yields  a  very  pessimistic  and  unlikely  duration  of  M=nine 
(9)  months  at  an  average  staff  level  of  P=  three  (3)  people.  As  a  result,  a  new  mathematical 
function  is  used  to  calculate  (predict)  the  calendar  months  for  a  given  amount  of  effort:  the 
function  is  only  radically  different  in  low  (under  25)  person-month  efforts  where  it  seems  more 
normal  to  have  an  equal  number  people  and  months  to  accomplish  the  task.  At  the  higher 
(greater  than  64)  person-month  efforts,  the  traditional  COCOMOII-1998  function  is  used  which 
is  based  on  the  traditional  cube-root-like  function  of  effort.  A  smooth  curve  is  fit  within  these 
ranges.  An  example  using  the  square  root  and  classic  3-times-cube-root  formulas  is  shown  in 
Figure  5. 


Figure  5.  CORADMO  Revision  to  COCOMO  II  Schedule  Estimator 
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E.2.3.3.  COCOMO  RAD  MODEL  (CORADMO) 

The  COCOMO  RAD  model  has  its  roots  in  the  results  of  a  1997  CSE  Focused  Workshop 
on  Rapid  Application  Development  [Boehm  -  Chulani  -  Egyed,  1997].  RAD  is  taken  to  mean 
application  of  any  of  a  number  of  techniques  or  strategies  to  reduce  software  development  cycle 
time.  A  "RAD  Opportunity  Tree"  presented  at  the  workshop  identified  5  classes  of  strategies 
whose  degree  of  implementation  can  be  used  to  parameterize  a  schedule  estimate  given  an  effort 
estimate  produced  by  COCOMOII-1998.  These  strategies  replace  the  simple  add-staff  strategy 
embodied  in  the  COCOMO  II  SCED  cost  driver.  The  five  classes  are  development  process  re¬ 
engineering  (DPRS),  re-use  and  very  high  level  languages  (RVHL),  collaboration  efficiency 
(CLAB),  architecture  and  risk  resolution  (RESL),  and  pre-positioning  of  assets  (PPOS).  RESL 
corresponds  to  the  COCOMO  II  scale  factor;  die  other  four  are  new.  All  have  their  effects 
reflected  as  multipliers  on  effort  (person  months,  PM),  schedule  (months,  M)  and/or  average 
number  of  personnel  (P).  Person  months  of  effort  can  actually  be  increased  because  certain  pro¬ 
active  strategies,  like  pre-positioning  of  assets,  are  only  possible  with  extra  effort. 

CORADMO  utilizes  the  COSSEMO  extension  which  allocates  effort  and  schedule  to  the 
stages  which  are  anchored  at  the  LCO/LCA/IOC  points  in  a  development  life  cycle.  Staged 
schedule  and  effort  distribution  is  needed  because  the  effects  of  the  RAD  strategies  identified 
above  is  different  for  the  different  stages. 

The  intent  of  CORADMO  is  to  calculate/predict  the  schedule  (months,  M),  personnel  (P), 
and  adjusted  effort  (person-months,  PM)  based  on  the  distribution  of  effort  and  schedule  to  the 
various  stages,  and  impacts  of  the  selected  RAD  strategy  schedule  driver  ratings  on  the  M,  P,  and 
PM  of  each  stage. 
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E.2.4  Evaluation  Cases  Considered 
E.2.4.1  EHART 

It  is  important  to  note  that  the  embedded,  high-assurance,  real-time  (EHART) 
applications  domain  is  not  the  only  domain  to  which  this  analysis  applies.  The  EHART  domain 
is  just  one  example  of  a  class  of  domains  called  “warfighting”  domains  in  the  NRC  “Ada  and 
Beyond”  report  [NRC,  1997]  for  which  commercial  technology  solutions  are  unlikely  to  satisfy 
many  of  DoD’s  application  needs. 

Other  examples  of  warfighting  domains  from  the  NRC  report  are  electronic  warfare, 
wideband  surveillance,  battle  management,  and  battlefield  communications.  The  assessment  of 
commercial,  KBSA,  and  EDCS  technology  impact  presented  in  this  report  for  the  EHART 
domain  would  be  applicable  in  large  measure  for  the  other  warfighting  domains  as  well.  For 
DoD  commercially-dominated  domains  such  as  finance  and  logistics  the  impact  of  commercial 
technology  would  be  considerably  higher,  and  the  impact  of  KBSA  and  EDCS  technology 
considerably  lower. 

The  particular  representative  EHART  application  used  in  this  analysis  is  a  100,000 
source  line  of  code  missile  software  suite  (lOOKSLOC;  1400  function  points  if  using  Ada  and 
the  71  SLOC/FP  ratio  for  Ada  in  [Jones,  1990]).  It  is  starting  its  development  in  1998.  The 
application  has  a  higher  required  reliability  (RELY*),  execution  time  constraint  (TIME^),  and 
main  storage  constraint  (STOR^)  than  the  COCOMO  II  1990’s  database  averages;  these  values 
have  been  adjusted  upwards  for  the  analysis. 

The  analysis  assumes  that  the  coimterpart  missile  software  projects  starting  in  2006  and 
2013  have  substantially  the  same  functionality,  but  their  accuracy  and  performance  requirements 
are  adjusted  upwards  to  capitalize  on  improvements  in  computer,  sensor,  and  guidance 
technology.  The  net  effect  of  these  on  key  COCOMO  II  cost  drivers  is  the  following: 

•  SIZE:  the  application  and  architecture  remain  similar  enough  to  enable  significant  gains 

via  domain-engineered  reuse  and  applications  generators. 


1  Required  Software  Reliability  (RELY):  This  is  the  measure  of  the  extent  to  which  the  software  must  perform  its 
intended  fiinction  over  a  period  of  time.  If  the  effect  of  a  software  failure  is  only  slight  inconvenience  then  RELY  is 
low.  If  a  failure  would  risk  human  life  then  RELY  is  very  high. 

2  Execution  Time  Constraint  (TIME):  This  is  a  measure  of  the  execution  time  constraint  imposed  upon  a  software 
system,.  The  rating  is  expressed  in  terms  of  the  percentage  of  available  execution  time  expected  to  be  used  by  the 
system  or  subsystem  consuming  the  execution  time  resource.  The  rating  ranges  from  nominal,  less  than  50%  of  the 
execution  time  resource  used,  to  extra  high,  for  95%  of  the  execution  time  resource  is  consumed. 

^  Main  Storage  Constraint  (STOR):  This  rating  represents  the  degree  of  main  storage  constraint  imposed  on  a 
software  system  or  subsystem.  The  rating  ranges  from  nominal,  less  that  50%,  to  extra  high,  95%. 
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•  CPLX:  the  application  complexity  level  remains  essentially  constant 

•  TIME,  STOR:  hardware  advances  reduce  the  resource  constraints,  but  do  not  eliminate 

them  due  to  the  increased  accuracy  and  performance  requirements. 

These  assumptions  are  also  applicable  to  applications  in  the  other  warfighting  domains  discussed 
above. 

E.2.4.2  KBSA 

The  analysis  separately  considers  the  application  of  the  two  major  KBSA  technology 
branches  -  Applications  Generation  (KG)  and  Decision  Support  (KD)  -  and  their  combined 
KBSA  application  (K).  The  analysis  assumes  that  programs  have  been  put  in  place  to  develop, 
extend,  and  transition  the  indicated  technologies,  with  a  particular  emphasis  on  the  development 
and  use  of  solutions  capitalizing  on  domain  knowledge  for  the  selected  DoD  applications  area  (in 
this  case,  EHART  missile  applications,  but  comparably  applicable  to  other  DoD  warfighting 
domains). 

The  KBSA  Applications  Generation  (KG)  program  would  focus  on  domain  models, 
domain  architectures,  tailorable  components,  domain-oriented  very  high  level  languages  for 
visual  programming  of  applications,  and  general  and  domain-oriented  applications  generator 
technology  enabling  efficient  generation  of  both  originally-specified  applications  or 
modifications  to  such  applications. 

The  KBSA  Decision  Support  (KD)  program  would  focus  on  general  and  domain-oriented 
technology  providing  effective  guidance  on  such  software  engineering  decisions  as  process 
model  selection;  domain  architecture  selection;  architecture  view  integration;  achievement  of 
system  properties  such  as  performance,  reliability,  safety,  survivability,  security,  interoperability, 
adaptability,  usability,  etc.;  risk  assessment  and  risk  management  planning;  COTS  integration; 
cost  and  schedule  management;  verification  and  test  planning  and  management.  The  intent 
would  be  to  bring  decision  support  for  such  decisions  to  the  level  of  maturity  and  applicability  as 
it  currently  is  for  such  decisions  as  change  control  or  process  control  via  control  limits. 

E.2.4.3  EDCS 

The  analysis  assumes  that  a  strong  EDCS  Phase  II  effort  would  be  put  in  place,  with  a  particular 
emphasis  on  developing  and  transitioning  domain-oriented  solutions  in  the  selected  warfighting 
domains.  It  also  assumes  that  a  strong  EDCS  follow  on  technology  development  effort  would  be 
put  in  place  to  expand  the  capabilities  being  developed  in  such  areas  as  requirements 
determination,  architecture,  software  collaboration,  incremental  specification/generation/ 
verification,  and  achievement  of  high-assurance  software. 

E.2.4.4  Commercial  Technology 

The  analysis  assumes  that  commercial  software  technology  will  continue  to  focus  on 
mass-market  domains  and  tool  support  such  as  broad-use  requirements,  design,  programming, 
test,  project  management,  and  configuration  management  aids.  As  indicated  in  the  [NRC,  1997] 
study,  commercial  software  tool  vendors  will  focus  primarily  on  technology  investments  with 
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near-term  payoff.  Thus,  commercial  tools  for  niche  markets  such  as  DoD  warfighting 
applications  will  develop  slowly. 

E.2.4.5  Technology  Combinations 

The  analysis  begins  with  the  1998  baseline.  It  uses  the  100  KSLOC  EHART  missile 
application  and  the  average  1990’s  cost  driver  ratings  from  the  COCOMO  II  database.  These 
ratings  might  be  considered  optimistic,  as  they  come  from  organizations  sufficiently  advanced  to 
have  collected  thorough  data  on  their  software  projects.  However,  this  is  offset  by  the  fact  that 
the  average  date  of  the  projects’  completion  is  1994.  On  balance,  the  ratings  should  be  well- 
representative  of  1998  EHART  project  practice,  with  the  EHART  adjustments  noted  for  the 
RELY,  TIME,  and  STOR  cost  driver  ratings. 

The  commercial  technology  and  DoD  practice  (CD)  ratings  for  2006  and  2013  are 
assessed  next.  The  KBSA  Applications  Generation  (KG)  and  Decision  Support  (KD)  ratings  for 
2006  and  2013  are  assessed  relative  to  the  CD  rating  levels,  followed  by  a  similar  assessment  for 
the  combined  KBSA  technology  impact  ratings.  Finally,  a  similar  set  of  ratings  for  2006  and 
2013  EDCS  technology  impact  is  assessed,  followed  by  ratings  for  the  combination  of  KBSA 
and  EDCS  technology. 

The  assessments  are  made  with  the  aid  of  a  spreadsheet  implementation  of  the  technology 
impact  analysis  model.  This  enabled  us  to  easily  perform  sensitivity  analyses  with  respect  to  our 
technology  ratings.  We  provide  one  set  of  sensitivity  analyses  indicating  the  results  of  our 
assessments  if  KBSA  and  EDCS  technology  were  optimistic  by  a  factor  of  2  with  respect  to 
commercial  technology.  Even  with  this  conservative  assessment,  the  results  indicate  a  strong 
DoD  payoff  for  the  technology  investments. 

E.2.5  Example  of  Model  Driver  Analysis 

We  presented  a  simple  summary  of  our  model  driver  analysis  approach  for  the 
COCOMO  II  TOOL  cost  driver  in  Section  E.2.2.  Figure  6  below  presents  an  example  of  the 
actual  model  driver  analyses  performed  for  this  study,  using  the  COCOMO  II  Architecture  and 
Risk  Resolution  (RESL)  scale  factor.  The  full  set  of  model  driver  analysis  worksheets  are 
provided  in  Section  L. 
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Figure  6.  RESL:  Architecture/  Risk  Resolution  Analysis 


RESL  and  other  COCOMO  II  scale  factors  add  increments  to  the  exponent  B  in  the 
COCOMO II  effort  equation: 

PM  =  A  n  (Effort  Multipliers)  (Size)® ,  where 

B  =  Bo+  0.01  S  (Scale  Factors). 

The  figures  by  the  triangles  show  the  calibrated  scale  factors  for  the  RESL  ratings, 
ranging  fi’om  0  for  very  thorough  architecture  and  risk  resolution  to  7.07  for  very  lax  architectme 
and  risk  resolution.  The  effect  of  a  scale  driver  is  higher  on  larger-size  products:  for  a  100 
KSLOC  product,  it  is  a  factor  of  100*  ”’°VlOO  =  1.38;  for  a  1000  KSLOC  product,  it  is  a  factor  of 
1000*  **’**  /lOOO  =  1.63.  The  behavioral  explanation  for  the  difference  is  that  projects  with  lax 
architecture  and  risk  resolution  spend  much  more  effort  in  rework  due  to  late  risk  and  defect 
removal  and  communications  overhead  across  poorly-defined  interfaces.  These  effects  are  larger 
for  larger  products. 

The  1998  baseline  average  from  the  COCOMO  II  project  database  is  a  RESL  scale  factor 
of  3.97.  The  effect  of  commercial  technology  and  DoD  practices  (CD)  on  this  scale  factor  was 
assessed  as  significant:  a  reduction  to  3.5  by  2006  and  to  3.0  by  2013.  This  is  due  partly  to 
ongoing  commercial  and  DoD  open  systems  initiatives,  such  as  CORBA  and  the  Defense 
Information  Infrastructure  Common  Operating  Environment  (DII-COE);  improved  commercial 
object-oriented  architecture  support  largely  based  on  the  Unified  Modeling  Language;  and  DoD 
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emphasis  on  risk  management  in  DoDD  5000.1,  DoDI  5000.2,  and  the  DoD  Software  Best 
Practices  Initiative. 

However,  the  effect  is  not  larger  because  these  advances  do  not  fully  address  a  number  of 
EHART  domain  issues  (e.g.,  distributed  real-time  deadline  management).  Thus,  there  are  a 
number  of  added  advances  that  can  be  achieved  via  KBSA  and  EDCS  technology  improvements, 
particularly  if  pursued  within  an  EHART  domain  context. 

The  KBSA  Applications  Generator  (KG)  technology  when  pursued  in  an  EHART 
domain  context  would  provide  improved  domain  architectures  and  avoidance  of  various  classes 
of  architecture  and  real-time  scheduling  risks.  The  KBSA  Decision  Support  (KD)  technology 
would  provide  complementary  improvements  via  decision  aids  for  such  issues  as  architecture 
choices,  risk  identification  and  resolution,  and  choices  of  scheduling  algorithms,  process  models, 
and  verification  strategies.  Each  of  these  is  assessed  as  providing  significant  additional  gains 
down  to  a  scale  factor  of  3.20  in  8  years  (2006)  and  down  to  2.50  in  15  years  (2013).  The 
sources  of  gains  are  complementary,  so  that  the  effect  of  a  2-prong  KBSA  program  (K)  is 
assessed  to  reduce  the  scale  factor  to  3.0  by  2006  and  2.2  by  2013. 

The  EDCS  program  (E)  would  provide  some  of  the  same  improvements,  particularly  via 
its  EHART  domain  coverage.  It  would  not  cover  some  of  the  KBSA  Decision  Support 
technology,  but  would  provide  stronger  capabilities  than  the  KBSA  program  in  such  areas  as 
general  architecture  description  and  analysis,  high-assurance  test  and  verification,  risk  reduction 
via  rationale  capture,  and  evolutionary/incremental  specification,  development,  and  verification 
techniques.  The  net  improvement  in  the  REST  scale  factor  over  the  CD  values  would  be  about 
the  same  as  those  for  a  ftill  KBSA  program,  to  3.0  by  2006  and  2.2  by  2013. 

Since  the  KBSA  and  EDCS  programs  provide  some  complementary  benefits,  there  is  an 
additional  scale  factor  reduction  assessed  for  a  combined  KBSA-EDCS  program  (EK)  down  to 
2.70  in  2006  and  1.70  in  2013.  The  overall  improvement  contribution  in  reducing  the  REST 
scale  factor  from  3.97  to  1.70  is  about  12%  for  a  100  KSLOC  product. 

E.2.6  Spreadsheet  Model  Overview 

A  multi-sheet  Excel  Workbook  has  been  developed  to  show  the  impacts  of  the 
COCOMOII  and  CORADMO  drivers  projected  over  time  and  technology  type  on  a  selected 
domain's  typically  sized  application.  This  spread  sheet  model  has  the  name  "Software 
Technology  Impact  Analyzer".  The  sheets  include  a  description  of  the  other  sheets  and  the 
COCOMOII- 1998  Calibration  values  and  ranges  for  reference,  and  sheets  for  the  COCOMOII 
and  CORADMO  drivers  and  their  impacts. 

The  workbook  also  has  several  protected  sheets  which  are  used  for  the  detailed  layout  of 
the  drivers  to  facilitate  the  graphing  shown  in  the  “Drivers”  sections.  There  are  also  protected 
sheets  for  the  default  values  (i.e.  the  USC  Center  for  Software  Engineering  assessed  values)  of 
the  COCOMO-II.1998  and  CORADMO  drivers. 
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E.2.6.1  COCOMO-II  Drivers,  Calculations  and  Impacts 

There  are  three  sheets  in  this  grouping.  The  first,  "CII  SF&EM  Drivers",  has  the 
projected  drivers  over  time.  The  second,  "CII  SF&EM  Data",  aggregates  the  driver  data  and 
does  the  COCOMOII  calculations.  The  third,  "CII  E&S  Impact",  has  graphs  showing  the  effort 
and  schedule  impact  of  the  COCOMO-II.  1 998  drivers  projected  over  time. 

"CII  SF&EM  Drivers"  shows  our  assessed  values  for  each  of  the  drivers  projected  over 
time  and  our  rationale,  and  allowing  the  input  of  new  values  and  additional  or  modified 
rationales.  A  graph  of  the  current  values  of  the  driver  projected  over  time  is  included;  the  data 
points  on  this  graph  change  when  new  values  are  entered. 

"CII  SF&EM  Data"  has  the  assessed  COCOMO-II.  1998  drivers,  both  scale  factors  and 
effort  multipliers,  organized  in  a  compact,  single  page  sheet  along  with  the  calculations  of  the 
COCOMOII  effort  and  schedule.  The  calculations  use  the  COCOMO-II.  1998  model  equations 
for  effort  and  the  COSSEMO  equations  for  schedule  (different  schedule  formulas  for  three 
ranges  of  months:  0  to  16;  16  to  64;  and  64  and  up).  Each  data  column  of  the  table  performs  the 
full  set  of  COCOMO-II  calculations  for  a  particular  year  and  one  technology-type  combination. 

E.2.6. 1 . 1  Example  Spreadsheet  Calculations 

An  example  of  the  comparative  COCOMO  II  effort  (PM)  calculations  is  shown  in  Table 
1.  The  columns  correspond  to  the  1998  baseline  and  pairs  of  2006  and  2013  projections  of  the 
effort  effects  of  commercial/general  DoD  technology  (CD);  the  KBS  A  Applications  Generator 
(KE),  Decision  Support  (KD),  and  Combined  (K)  effects;  the  EDCS  (E)  effects;  and  the 
combined  EDCS  and  KBS  A  (EK)  effects. 

The  six  rows  starting  with  row  4  (PREC  -  row  7  in  the  computer  version)  show  the 
assessed  values  of  the  five  COCOMO  II  scale  factors  and  their  sum  S.  For  example,  the  values 
for  Architecture  and  Risk  Resolution  (REST)  are  exactly  those  explained  in  Figure  6.  Row  10 
shows  the  impact  of  the  various  technologies  in  reducing  the  COCOMO  II  diseconomy-of-scale 
factor  B  from  its  1998  baseline  value  of  1.076  to  a  2013  EK  value  of  0.991.  Recall  that  the 
COCOMO  II  effort  estimation  equations  in  Section  E.2.5  are: 

PM  =  ATI  (Effort  Multipliers)-(Size)®  ,  where 

B  =  Bo+  0.0 1-Z  (Scale  Factors) 

For  COCOMO  11.1998,  the  calibrated  values  of  A  and  Bo  are  2.94  and  0.91,  respectively. 

Rows  11-27  (14-30  in  computer  version)  of  Table  1  show  the  corresponding  assessed 
effects  of  the  various  technologies  on  the  COCOMO  II  effort  multipliers.  For  example,  the 
reduction  in  the  TOOL  effort  multiplier  from  a  1998  baseline  value  of  1.01  to  a  2013  value  of 
0.86  for  the  combination  of  KBS  A  and  EDCS  (EK)  technology  is  exactly  as  discussed  in  Section 
E.2.2.  The  resulting  effect  of  the  ensemble  of  effort  multipliers  is  to  reduce  their  product  n  (row 
28)  from  a  1998  baseline  of  1.21  to  a  2013  EK  value  of  0.38,  a  factor  of  more  than  3. 
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Table  1.  Comparative  COCOMO II  Effort  Calculations 


The  other  major  influence  on  software  effort  in  the  COCOMO  II  equations  is  SIZE, 
roughly  equivalent  to  the  number  of  source  instructions  one  needs  to  program.  Technology  can 
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reduce  this  via  reuse,  reduced  breakage,  or  very  high  level  languages,  particularly  m  concert  with 
domain  specialization  and  knowledge.  The  SIZE  worksheet  in  Section  L  provides  the  rationale 
for  the  size  reduction  effects  of  the  various  technologies;  the  resulting  reductions  are  shown  in 
row  35  of  Table  1. 

Given  the  values  of  B,  U,  and  SIZE,  the  COCOMO  II  equations  produce  the 
corresponding  effects  on  effort  (CII_PM)  of  the  various  technologies  (row  36).  The  effects  will 
be  displayed  graphically  and  discussed  in  Section  E.3  below.  The  full  set  of  spreadsheet 
calculations  for  cost/effort  and  schedule  impacts  are  provided  on  Section  L. 

E.2.6.2  CORADMO  Drivers,  Calculations  and  Impacts 

Like  the  COCOMOII-1998  sheets,  there  are  three  sheets  in  this  grouping.  The  first, 
"RAD  Drivers  ",  has  the  projected  drivers  over  time.  The  second,  "  CORADMO  Data  ", 
aggregates  the  driver  data  and  does  the  CORADMO  calculations.  The  third,  RAD  SM  Impact  , 
hL  graphs  showing  the  resulting  impacts  of  the  COCOMO-II.1998  and  CORADMO  drivers 
projected  over  time. 

"RAD  Drivers"  has  our  assessed  values  for  each  of  the  CORADMO  schedule  multiplier 
drivers  projected  over  time  and  our  rationale,  and  allowing  the  input  of  new  values  and 
additional  or  modified  rationales.  A  graph  of  the  current  values  of  the  driver  projected  over  time 
is  included;  the  data  points  on  this  graph  change  when  new  values  are  entered. 

"CORADMO  Data"  aggregates  the  CORADMO  drivers,  both  scale  factors  and  effort 
multipliers,  organized  in  a  compact,  single  page  sheet  along  with  the  c^culations  of  the 
CORADMO  effort  and  schedule.  The  sheet  also  allows  for  the  distribution  of  effort  and 
schedule  to  the  various  stages  as  specified  in  the  COSSEMO  model.  The  calculations  use  the 
CORADMO  model  equations  to  distribute  schedule  and  effort  based  on  the  selected  percentage 
allocations  and  the  schedule  multiplier  driver  ratings.  Each  PM  &  M  pair  of  data  columns  of  the 
table  performs  the  full  set  of  CORADMO  calculations  for  a  particular  year  and  one  technology- 
type  combination. 

"RAD  SM  Impact"  has  graphs  showing  the  effort,  schedule  and  Full-time  Software 
Personnel  (FSP)  impacts  of  the  entered  CORADMO  drivers  projected  over  time.  All  three  are 
shown  for  each  stage  (Inception,  Elaboration,  and  Construction)  since  the  schedule  multipliers 
can  impact  both  effort  and  schedule;  and  the  FSP  values  are  then  simply  the  result  of  dividing 
effort  (in  person  months)  of  a  stage  by  its  duration  (in  months). 

E.2.6.3  Technical  Impact  Final  Results 

At  the  end  of  the  "RAD  SM  Impact"  worksheet,  following  the  RAD  impacts  by  stage,  are 
the  summary  charts  for  effort  and  schedule  by  technology  over  time  that  result  from  the 
COCOMO  II  and  CORADMO  driver  changes  over  time.  The  data  for  these  charts  is  actually 
shown  on  the  second  page  of  the  "CORADMO  Data"  sheet.  The  effort  and  schedule  results  are 
generated  by  adding  the  effort  or  schedule,  respectively,  for  all  three  stages.  Since  COCOMO  II 
only  calculates  the  effort  and  schedule  for  development,  a  second  set  of  summary  charts  was 
generated  so  the  COCOMO-II  model  results  could  be  easily  compared  to  the  CORADMO  model 
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results.  The  second  set  of  charts  totals  eifort  and  schedule  only  for  the  Elaboration  and 
Construction  stages.  Along  with  each  chart  are  copies  of  the  rows  of  the  appropriate  data  from 
"CORADMO  Data"  sheet. 

E.3  KBSA  Comparative  Impact  Analysis  Results 
E.3.1  Cost/Effort  Impact 

Figure  7  shows  the  relative  assessed  effect  of  the  various  technologies  in  reducing  the 
amount  of  effort  required  to  develop  the  100  KSLOC  EHART  missile  application.  The  1998 
baseline  effort  is  505  person  months  (PM).  The  assessed  effect  of  commercial  technology  and 
DoD  practices  (CD)  is  to  reduce  this  effort  by  a  factor  of  2.5  (to  204  PM)  by  2006  and  another 
factor  of  3  (to  64  PM)  by  2013. 


Figure  7.  Impact  of  Technologies  on  Software  Effort  or  Cost 


These  are  impressive  gains,  but  it  must  be  remembered  that  most  of  them  are  also  within 
reach  of  ^y  rogue  nation  or  terrorist  organization  purchasing  commercial  technology.  To  keep  a 
competitive  military  edge,  USAF  and  DoD  need  to  have  superior  software  management  and 
technology  capabilities. 

The  remainder  of  the  curves  indicate  that  significant  improvements  are  available  via 
investments  in  KBSA  and  EDCS  technology.  A  combination  of  EDCS  and  KBSA  technology 
investments  is  assessed  to  provide  another  factor-of-3  gain  over  commercial/DoD  technology 
and  practices  (to  65  PM)  by  2006  and  another  factor-of-6  gain  (to  10  PM)  by  2013. 
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An  analysis  of  the  major  factors  contributing  to  the  additional  technology  gains  indicates 
that  the  primary  source  is  the  reductions  in  effective  size  due  to  combinations  of  software 
technology  and  domain  knowledge  engineering.  These  account  for  almost  all  of  the  gains  from 
KBSA  Applications  Generator  (KG)  technology,  and  much  of  the  gains  from  EDCS  (E) 
technology. 

However,  as  seen  by  the  relative  position  of  the  KG  effort  values,  domain  engineering  is 
not  a  single  silver-bullet  solution  for  achieving  DoD  warfighting  software  productivity  gains  (nor 
is  commercial  technology).  Other  particularly  strong  contributors  were: 

•  High  assurance  technologies,  strongly  improving  the  cost  driver  fectors  for  RESL 
(Architecture  and  Risk  Resolution),  RELY  (Required  Reliability),  TIME  (Execution 
Time  Constraints),  and  TOOL  (Use  of  Software  Tools); 

•  Reauirements,  collaboration,  and  rationale  capture  technologies,  strongly  improving 
the  cost  of  driver  factors  for  RESL  (Architecture  and  Risk  Resolution),  TEAM  (Team 
Cohesion),  PCON  (Personnel  Continuity),  and  SITE  (Multisite  Development); 

•  KBSA  decision  support  technologies,  strongly  improving  the  cost  driver  factors  for 
RESL  (Architecture  and  Risk  Resolution),  TEAM  (Team  Cohesion),  ACAP  and 
PCAP  (Analyst  and  Programmer  Capability),  PCON  (Personnel  Continuity),  and 
TOOL  (Use  of  Software  Tools),  particularly  in  the  longer  term  (2013). 

•  Software  architecture  and  incremental  evolution  support  technologies,  strongly 
improving  the  cost  driver  factors  for  RESL  (Architecture  and  Risk  Resolution), 
TOOL  (Use  of  Software  Tools),  and  MCF  (Maintenance  Change  Fraction,  factored 
into  SIZE). 

Some  software  technology  areas  receiving  recent  DoD  support  which  did  not  generate 
significant  cost  and  schedule  impacts  over  commercial  technology  impacts  were  Integrated 
Software  Environments  and  Formal  Methods. 

Integrated  Software  Environments  are  being  supported  at  such  a  critical-mass  level  by 
commercial  technology  that  they  become  too  difficult  to  match  via  independent  DoD  efforts. 
Formal  Methods  are  very  important  for  ultra-high  assurance  DoD  applications  such  as  computer 
security  and  information  warfare;  they  also  provide  support  for  small-scale  applications 
generation  capabilities.  But  they  do  not  apply  well  or  scale  up  well  to  large  COTS-intensive 
applications. 

E.3.2  Schedule  Impact 

Figure  8  shows  the  counterpart  results  of  the  CORADMO-based  analysis  of  the  impact  of 
the  various  technologies  on  the  development  schedule  for  the  representative  EHART  project. 
The  detailed  spreadsheet  analysis  for  this  and  related  effects  are  in  Section  L. 
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Figure  8.  Impact  of  Technologies  on  Software  Schedule 


The  gener^  trends  of  the  results  are  similar  to  those  in  the  cost/effort  analysis  shown  in 
Figure  7.  The  main  highlights  are: 

•  Conmercial  technology  and  DoD  practices  (CD)  provide  significant  gains,  reducing 
the  1998  baseline  EHART  schedule  from  24.1  months  to  16.5  months  by  2006  and  to 
10.3  months  by  2013.  Besides  the  contributions  via  reduced  effort,  the  primary  added 
RAD  contributions  came  from  added  collaboration  efficiencies  via  commercial 
collaboration  technology  gains  and  DoD  Integrated  Product  Team  practice  gains. 

•  KBSA  technology  has  the  potential  to  reduce  the  development  schedule  even  lower 
to  12.4  months  by  2006  and  to  3.5  months  by  2013.  Besides  reducing  effort,  the 
added  RAD  gains  come  from  rapid  prototyping  via  domain  knowledge-based 
application  generators  and  from  KB  decision  aids  for  accelerated  processes  and 
collaboration. 

•  EDCS  technology  can  reduce  the  schedule  even  further,  via  domain  application 

generators  plus  other  architecture-based  prepositioning  of  assets  and  via  collaboration 
technology. 

•  A  umfied  EDCS/KBSA/commercial  technology/DoD  practices  approach  is  assessed 
to  reduce  the  schedule  even  further,  to  10.3  months  by  2006  and  to  2.3  months  by 
2013.  The  main  reason  for  the  significant  additional  shortening  by  2013  is  that  the 
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project  size  has  been  reduced  to  the  point  that  schedule  and  staffing  can  follow  the 
square-root  relation  in  Figure  5  rather  than  the  3-times-cube-root  relation. 

The  CORADMO  analyses  also  address  effects  of  the  RAD  strategies  on  development 
effort,  but  the  net  results  for  the  EHART  project  were  not  much  different  from  the  COCOMO  II 
estimates.  The  detailed  results  are  in  Section  L. 

E.3.3  Sensitivity  Analyses 

The  technology-impact  results  in  Figure  7  and  Figure  8,  and  in  the  detailed  analyses  in 
Section  L,  are  based  on  the  KBSA  ADM  evaluation  and  the  authors’  experience-based 
judgement  It  is  always  possible  that  these  judgements  are  overly  optimistic  or  pessimistic.  To 
reduce  these  possibilities,  we  have  performed  or  provided  several  cross-checks: 

•  Boehm  and  Brown  independently  assessed  the  effects  of  the  technologies  on  the 
model  parameters,  resulting  in  considerable  iteration  of  the  parameter  values. 

•  We  reviewed  the  values  and  results  with  two  industry  software  metrics  experts  and 
did  a  preliminary  briefing  of  the  results  for  AFRL/IFTD,  resulting  in  further 
suggestions  and  iteration  of  the  results. 

•  We  have  done  a  comparative  analysis  with  the  results  of  similar  studies  (in  Section 
E.3.4  below),  and  have  found  the  results  fairly  consistent. 

•  We  have  developed  the  spreadsheet  model  so  that  anyone  with  different  technology 
assessments  can  enter  them  in  the  model  and  see  the  results.  The  model  and  its  usage 
instructions  are  in  Section  N,  and  also  available  via  our  Web  site  at 

http;//sunset.usc.edu/COPROMO/KBSA_LCE/kbsa_lce.html. 

•  We  have  used  the  spreadsheet  model  to  perform  various  analyses  of  the  sensitivity  of 
the  results  to  the  model’s  inputs.  The  most  significant  example  is  presented  next; 
others  are  included  in  Section  L. 

Figure  9  shows  the  result  of  a  sensitivity  analysis  in  which  we  assumed  that  all  of  our 
assessments  of  the  influence  of  KBSA  and  EDCS  technology  were  optimistic  by  a  factor  of  2. 
For  example,  in  the  last  column  of  Table  1  (Section  E.2.6.1.1),  the  effect  of  the  combined  2013 
EDCS  and  KBSA  technology  was  to  reduce  the  scale  factor  sum  S  from  its  2013  CD  value  of 
12.2  down  to  8.1.  Similarly,  the  assessment  reduced  the  effort  multiplier  product  n  from  0.67  to 
0.38,  and  the  effective  size  from  30  to  10  KSLOC. 

For  the  analysis  in  Figure  9,  these  differences,  and  all  the  other  differences  between  the 
2013  CD  assessed  values  and  the  assessed  values  in  the  columns  to  the  right  of  it  were  reduced 
by  a  factor  of  2.  For  the  2013  EK  values  above,  the  E  value  became  10.2,  the  11  value  became 
0.52,  and  the  SIZE  value  became  20  KSLOC. 
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Figure  9.  Effect  of  50%  Reduction  in  Effort  Improvement  Factors 

The  results  in  Figure  9  show  that  reducing  the  EK  improvement  parameters  by  a  factor  of 
2  leads  to  reduction  in  effort  savings  by  a  factor  of  1 .7-1 .75  (e.g.,  for  2013,  (67-1 1)/(67-32)  = 

1.75).  For  the  other  technology  comparisons,  the  results  were  similar:  a  savings  reduction  close 
to  but  somewhat  less  than  a  factor  of  2.  Even  with  the  more  conservative  payoff  assessments, 
the  resulting  benefits  appear  well  worth  the  technology  investments. 

E.3.4  Comparison  to  Other  Studies 

A  good  comparative  study  of  commercial  technology  impact  on  software  effort  or 
productivity  is  provided  in  [Bernstein,  1997].  Bernstein’s  Figure  10  shows  the  trend  in 
“expansion  factor,”  the  ratio  of  machine  language  instructions  per  line  of  generated  source  code, 
between  1960  and  1995.  The  trend  line  in  Figure  10  indicates  a  factor-of-10  improvement  every 
20  years,  a  figure  very  consistent  with  this  study’s  factor-of-7.7  improvement  from  commercial 
technology  over  15  years.  Bernstein’s  1995  and  projected  2000  expansion  factors  rise  above  this 
trend  line,  and  appear  to  be  due  to  a^strong  degree  to  domain-oriented  reuse  gains.  This  is  again 
consistent  with  this  study’s  detenhination  of  additional  productivity  benefits  from  domain- 
oriented  technology  in  DoD  warfighting  domains  not  well  covered  by  commercial  technology. 
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Figure  10.  Software  Expansion  Factor  vs.  Time  [Bernstein,  1997] 

A  similar  study  to  this  was  done  for  the  DoD  STARS  program  [Boehm  -  Standish,  1983]. 
It  predicted  a  factor-of-4.34  effort  reduction  over  10  years,  but  did  not  distinguish  between 
STARS-based  and  commercial  technology-based  gains.  The  DoD  Software  Technology 
Strategy  [DoD,  1991]  used  a  different  value-chain  approach  to  estimate  effort  reductions  for  each 
type  of  software  activity  (requirements,  design,  code,  test,  documentation,  management,  etc.).  It 
indicated  a  potential  factor-of-12  savings  in  software  effort  over  15  years,  but  a  more  likely 
factor-of-4  savings  in  15  years  due  to  technology  transition  inertia.  It  did  not  distinguish 
between  savings  due  to  DoD  technology  investments  vs.  those  due  to  commercial  technology. 

Both  of  these  studies  indicated  that  effective  size  savings  via  reuse  was  the  largest  source 
of  savings,  although  only  the  latter  emphasized  domain-specific  reuse.  All  three  of  the  major 
recent  books  on  software  reuse,  [Poulin,  1996^  Jacobson  et  al.,  1997;  Reifer,  1997],  indicate  that 
reuse  is  the  most  attractive  source  of  software  savings,  and  that  domain-specific  reuse  is  far  more 
effective  than  domain-independent  reuse. 


E.4  Conclusions  and  Recommendations 

E.4.1  Conclusions 

1.  The  KBSA  Advanced  Development  Model  was  not  sufficiently  robust  and  scalable  to 
provide  appreciable  benefits  to  critical  DoD  software  projects.  We  found  its  framework 
difficult  to  scale  to  large  projects,  and  its  toolset  to  be  small  with  respect  to  large  project 
needs. 

2.  However,  the  KBSA  concepts,  if  otherwise  realized,  have  strong  potential  for 
reducing  costs  and  schedules  for  DoD-critical  warfighting  software  applications.  The  key 
to  realizing  this  potential  is  to  develop  the  KBSA  technology  in  the  context  of  DoD 
warfighting  domain  knowledge.  For  KBSA  Applications  Generation,  this  implies 
focusing  on  DoD  warfighting  areas  where  a  good  start  exists  towards  domain 
engineering,  domain  architecting,  and  development  and  use  of  reusable  domain  assets. 
Some  of  the  USAF/ESC  Product  Line  areas  are  good  candidates,  as  are  some  avionics, 
missile,  and  sensor  processing  domains. 

For  KBSA  Decision  Support,  this  involves  both  domain-specific  decision  aids  and  more 
generic  decision  aids  for  security,  survivability,  risk  assessment,  architecture  definition, 
and  process  definition  and  management. 

3.  The  KBSA  Decision  Support  area  provides  an  attractive  new  direction  for  Air  Force 
^d  DoD  software  technology  investments.  Knowledge-based  and  agent-based  mixed- 
initiative  technical  approaches  are  generating  high  payoffs  in  decision  support  areas 
similar  to  software  engineering  such  as  manufacturing  and  logistics.  A  similar  initiative 
for  software  engineering  would  build  on  existing  technology  in  attractive  new  directions. 

4.  EDCS  technology  investments  and  payoffs  have  both  commonalities  and 
complementarities  with  respect  to  KBSA  technology  investments  and  payoffs.  To  some 
extent,  good  portions  of  a  KBSA  Applications  Generation  program  have  been  pursued 
under  EDCS.  A  KBSA  Decision  Support  program  could  build  upon  and  provide  new 
directions  for  some  current  EDCS  efforts  in  such  areas  as  rationale  capture,  design  webs, 
and  architecture  technology. 

5.  The  Software  Technology  Impact  Analysis  spreadsheet  model  provides  a  useful  tool 
for  assessing  alternative  software  technology  assessment  strategies.  It  enables 
technology  managers  to  relate  candidate  technologies’  effects  on  software  cost  and 
schedule  drivers  to  resulting  project  cost  and  schedule  improvements.  It  is  well 
calibrated  and  baselined  with  respect  to  current  software  practice  and  commercial 
technology  and  productivity  trends. 

6.  There  is  no  software  technology  silver  bullet  for  DoD  warfighting  systems,  including 
reliance  on  commercial  technology.  A  mixed  strategy  of  commercial  technology  and 
complementary  DoD  warfighting  software  technology  is  necessary  to  address  DoD’s  full 
range  of  needs. 
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E.4.2  Recommendations 


1.  The  Air  Force  and  DoD  should  continue  their  investments  in  software  technology, 
particularly  for  warftghting  domains.  Both  a  strong  DoD  integrating  emphasis  and 
strong  service  domain-focused  applications  emphasis  are  needed.  A  sustaining  EDCS 
Phase  I-level  technology  generation  program  with  continuing  EDCS  Phase  Il-level 
technology  insertion  efforts  appears  to  be  an  appropriate  magnitude  for  such  a  program. 

Given  their  commonalties,  a  combined  EDCS-KBSA  program  could  be  pursued  at 
roughly  the  initially-planned  EDCS  Phase  I  and  II  levels  of  investment.  If 
complementary  DoD  management  and  commercial  technology  assimilation  strategies 
were  in  place,  the  program  could  generate  software  cost  and  schedule  benefits  similar  to 
those  estimated  in  the  analysis  in  Section  E.3.  As  noted  in  [NAE,  1995]  and  [Fox  et  al., 
1995]  the  resulting  retum-on-investment  ratio  for  DoD  would  be  very  high. 

From  Section  E.3.1,  the  strongest  software  technology  investment  impacts  are  likely  to 
come  from  DoD  warfighting  domain-based  reuse  and  applications  generation;  high 
assurance  technologies;  requirements,  collaboration,  and  rationale  capture  technologies, 
and  software  architecture  and  incremental  evolution  support  technologies.  Areas  with 
less  strong  general  investment  impacts  were  integrated  software  environments  and  formal 
methods,  although  formal  methods  are  important  for  some  small  or  ultra-high  assurance 
DoD  applications  areas. 

2.  The  software  technology  investments  should  be  part  of  an  integrated  DoD  software 
management/process/technology  strategy  to  ensure  DoD  software  supremacy, 
particularly  in  warftghting  domains.  Besides  the  technology  development  and  transition 
investments,  elements  of  the  integrated  strategy  are  OSD  management  initiatives  toward 
integrated  Capability  Maturity  Models  and  best  practices;  acquisition  initiatives  such  as 
the  USAF/ESC  Product  Line  strategy;  domain  engineering  initiatives  such  as  the  SEI 
Product  Line  Systems  program;  and  enterprise  architectures  building  on  the  DII-COE. 

The  conclusions  and  recommendations  are  not  particularly  surprising.  They  are 
quantitative  counterparts  of  the  qualitative  conclusions  and  recommendations  about  the  need  for 
DoD  investments  in  software  technology  in  recent  Air  Force  Scientific  Advisory  Board  studies 
[SAB,  1995;  SAB,  1997]  and  recent  DoD-level  studies  [NAE,  1995;  Fox  et  al,  1995;  NRC, 
1997].  What  is  surprising  is  that  in  response  to  this  consistent  set  of  signals,  DoD  has  been 
reducing  its  level  of  investments  in  software  technology. 
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F.  Discussion  of  Tools:  Technology  Impact  Analyzer 
F.l  Tool  Overview 

The  Technology  Impact  Analyzer  is  a  multi-sheet  Excel  Workbook  that  shows  the 
impacts  of  the  COCOMO  II  and  CORADMO  drivers  projected  over  time  and  technology-type 
on  a  selected  domain's  typically  sized  application.  The  sheets  include  an  overview  and  sheets  for 
the  COCOMO-II.1998  ,  COSSEMO  and  CORADMO  drivers,  data  and  their  impacts. 


The  overview  sheet  includes  abbreviations  and  descriptions  of  the  other  sheets  on  the  first 
page,  Figure  1 1 .  More  information  on  the  overview  and  the  other  sheets  is  available  Appendix  5. 


Teohonology  Impact  Analsjer  Abbreyiations  ^ 

. . i  Cp=J  Commercial  technology  and  DoD  general  practice 

KBSA=j  Knowledge  Based  Software  Assistant 


KG=  KBSA  Applications  Generators,  including  KB  domain  engineering  ('CD) 
i  KD=  I  KBSA  Project  Decision  Support  (SE  decision  assistant  crcncept)  (*00) 
K=:BothKQ  &KD  :  !  j 

E= ;  EDCS  or  Eyolutionary  Oeliuery  of  Complex  Software  Sysetms  i 

EK=  both  EDCS  &  KBSA  (KG  8e  KD) 

EHART= .  Embedded.  High  Assurance.  Real  Time  (baseiine  application  domain] 


The  T echorioiogy  Impact  Analyzer  yorkbook  has  several  worksheets  covering 
Technoloy  Impact  Analyzer  Overview  Sheet  ("TIA"  tab):  This  sheet  with 
:  1.  J  Abbreviations  and  worksheet  overviews 

2.  I  COCOMOil-  l  333  Calibration  values  and  ranges 
COCOMOII-1338  Scale  Factor Jk  Effort  Multiplier  Privets 


;  ClfciCoCoMo  11-1338  i 
SF=  Scaie  Factor  EM=  Effort  Muitiplier 
i  RAD= :  CoRADMo  (scheduie  &  effort) 
SM= ;  Scheduie  Multipiier 
PM=  Person  Months 
M=  Months 

I  FSP=  Fulltirne  Software  Personnel 
SSE=  Staged  Schedule  and  Effort 
l=lncepstion  E=  Elaboration  iCg  Construction 


J . 


COCOMOII-1 338  Scaje  Factor  &  Effort  M  Drivers  projected  over  time  ("Cll  priyers' 
.  Individual  parameters  displayed  with  default  and  new/current  numeric  yaiues.  and 
COCOMOii- 1 338  Scale  Factor  &  Effort  Multiplier  Data  C’cii  Data”  tab)  j 
1  :  Parameters  organized  in  a  cornpact,  single  page  for  review,  along  with  scheduie 
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Figure  1 1 .  TIA  Abbreviations  and  Sheet  Descriptions 


F.  1 . 1  COCOMO-II  Drivers,  Calculations  and  Impacts 

There  are  three  sheets  in  this  grouping.  The  first,  "CII  Drivers",  has  the  current  projected 
scale  factors  and  effort  multipliers  drivers  over  time  and  allows  for  changing  the  default  values 
to  their  new  values.  The  second,  "CII  Data",  aggregates  the  driver  data  and  does  the  COCOMO 
II  calculations.  The  third,  "CII  Impact",  has  graphs  showing  the  effort  and  schedule  impact  of 
the  COCOMO-II.1998  drivers  projected  over  time.  Only  “CII  Drivers”  allows  input.  More 
information  on  CII  Drivers”  is  available  the  next  section.  More  information  on  all  these  sheets 
is  available  in  Appendix  5. 
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F.  1 .2  CoSSEMo  Schedule  and  Effort  Percentage  Distributions  per  Stage 

This  sheet  “SSE  %”,  allows  the  input  of  percentage  distributions  of  effort  and  schedule 
to  the  various  stages.  Inception,  Elaboration,  and  Construction,  as  required  for  the  COCOMO  II 
Staged  Schedule  and  Effort  Model  (COSSEMO).  The  impact  of  these  distributions  on  the 
COCOMO-II.1998  baseline  results  is  shown  in  the  chart  at  the  end  of  the  worksheet. 

F.l  .3  CORADMO  Drivers,  Calculations  and  Impacts 


Like  the  COCOMO-II.1998  sheets,  there  are  three  sheets  in  this  grouping.  The  first, 
"RAD  Drivers",  shows  the  new  or  default  projected  drivers  over  time.  The  second,  '  RAD  Data  , 
aggregates  the  driver  data  and  does  the  CORADMO  calculations.  The  third,  "RAD  Impact  ,  has 
^aphs  showing  the  resulting  impacts  of  the  CORADMO  drivers  projected  over  time  when 
applied  to  the  corresponding  COCOMO-II.1998  results  with  the  COCOMO  dnvers  projected 
over  time.  At  the  end  of  the  page  of  the  "RAD  Data"  sheet  are  the  summa^  calculations  for 
totals  of  schedule  and  effort  across  stages  allowing  comparison  with  the  results 
COCOMO-II.1998.  Only  “RAD  Drivers”  allows  input.  More  information  on  RAD  Drivers  is 
available  the  next  section.  More  information  on  all  these  sheets  is  available  in  Appendix  5. 


F.l. 4  Technical  Impact  Final  Results 

At  the  end  of  the  "RAD  Impact"  worksheet,  following  the  nine  RA.D  impacts  by  stage 
charts,  are  the  summary  charts  for  effort  and  schedule  by  technology  over  time  that  result  from 
the  COCOMO-II.1998  and  CORADMO  driver  changes  over  time.  The  effort  and  schedule 
results  are  generated  by  adding  the  effort  or  schedule,  respectively,  for  either  all  toee  stages  or 
just  for  the  Elaboration  and  Construction  stages.  More  information  on  this  sheet  is  available  m 

Appendix  5. 
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F.2  COCOMO II  Drivers  Display,  Modification  and  Rationale 

The  worksheet  with  the  "CII  Drivers"  shows  all  of  our  assessed  values  for  each  of  the 
scale  factor  or  effort  multiplier  drivers,  projected  over  time  and  technology,  and  our  rationale. 
Each  page  of  this  worksheet  has  the  current  projected  COCOMO-II.1998  drivers,  both  scale 
factors  and  effort  multipliers,  over  time  and  allows  for  changing  the  default  values  to  their  new 
values.  The  rationales  for  the  default  settings  of  the  drivers  are  included;  they  should  be 
modified  when  “new”  values  are  provided.  Figure  12  shows  the  scale  factor  PREC’s 
information. 


Figure  12.  PREC’s  Driver  Entry,  Modification  and  Display 


The  default  and  current  values  of  the  driver,  projected  over  time  and  technology,  are  shown  in  a 
small  table  above  the  chart  of  the  current  values.  The  last  row  of  this  table  accepts  the  input  of 
new  values  of  the  driver,  projected  over  time  and  technology.  The  chart  below  the  table  shows 
the  driver’s  current  values  over  time  for  each  technology  combination.  The  data  points  on  this 
graph  change  when  new  values  are  entered. 

Since  each  value  of  a  driver  should  have  a  rationale,  the  rationales  for  the  default  values 
(our  assessed  values)  are  shown  below  the  chart.  The  area  below  the  rationales  for  the  default 
values  allows  the  input  of  additional  or  modified  rationales. 
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F.3  COSSEMO  Distribution  of  Schedule  and  Effort  per  Stage 

There  are  two  parts  to  this  worksheet:  1)  Input  of  inception,  elaboration  and  construction 
stages'  schedule  and  effort  percentages;  and  2)  Chart  of  distribution  of  schedule  and  effort 
impacts  on  the  current  COCOMO II  calculations. 

Input  of  schedule  and  effort  percentage  distributions  per  stage,  Inception,  Elaboration, 
and  Construction,  is  required  for  the  COCOMO  II  Staged  Schedule  and  Effort  Model 
(COSSEMO).  To  help  visualize  these  distributions,  their  impact  on  the  COCOMO-II.1998  lOOK 
EHART  baseline  is  displayed  in  the  chart  at  the  end  of  the  worksheet.  Figure  13  shows  the 
entire  content  of  this  worksheet. 
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Figure  13.  Staged  Schedule  and  Effort  Distribution 


The  values  of  the  Inception  and  Elaboration  percentages  for  schedule  and  effort  are 
adjusted  by  clicking  on  the  up/down  arrows  (spiimers)  shown  to  the  right  of  their  values.  The 
current  values  are  displayed  in  bold,  along  with  the  corresponding  calculated  values  for  the 
Construction  stage.  The  default  values  for  all  the  percentages  are  shown  in  italics. 


F.4  CORADMO  Drivers  Display,  Modification  and  Rationale 


"RAD  Drivers"  has  our  assessed  values  for  each  of  the  relevant  CORADMO  schedule 
and  effort  multipliers  projected  over  time  and  our  rationale.  It  also  allows  the  input  of  new 
values  and  additional  or  modified  rationales.  A  graph  of  the  current  values  of  each  driver 
projected  over  time  and  technology  is  included;  the  data  points  on  this  graph  change  when  new 
values  are  entered. 
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Figure  14.  RVHL’s  Inception  Stage  Schedule  Multiplier  Driver  Information 


The  default  and  current  values  of  the  driver,  projected  over  time  and  technology,  are 
shown  in  a  small  table  above  the  chart  of  the  current  values.  The  last  row  of  this  table  accepts 
the  input  of  new  values  of  the  driver,  projected  over  time  and  technology.  The  chart  below  the 
table  shows  the  driver’s  values  over  time  for  each  technology  combination. 

Since  each  value  of  a  driver  should  have  a  rationale,  the  rationales  for  the  default  values 
(our  assessed  values)  are  shown  below  the  chart.  The  area  below  the  rationales  for  the  default 
values  allows  the  input  of  additional  or  modified  rationales. 
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F.5  Final  Results:  Technology  Impacts 

At  the  end  of  the  "RAD  Impact"  worksheet,  following  the  nine  RAD  impacts  by  stage 
charts,  are  the  summary  charts  for  effort  and  schedule  by  technology  over  time  that  result  from 
the  COCOMO-II.1998  and  CORADMO  driver  changes  over  time.  The  “new/current”  data  for 
the  summary  charts  is  actually  shown  at  the  end  of  the  "RAD  Data"  sheet. 

There  are  three  different  types  of  charts; 

1.  Overall  (effort  or  schedule  for  all  three  stages  or  just  for  development  (elaboration  plus 
construction),  with  some  of  these  having  alternative  axes  layouts, 

2.  COCOMO-II.1998  compared  to  CORADMO  (final)  results,  with  some  of  these  charts 
showing  only  the  major  technology  groupings  (CD,  K  and  EK); 

3 .  Final  results  of  default  driver  settings  compared  to  new/current  driver  settings’  results. 

The  list  of  all  the  charts  corresponding  to  final  results  is  shown  in  the  Table  2,  below. 
More  detailed  information  and  examples  of  these  charts  are  given  in  an  Appendix. 


Number 

Title 

1. 

CORADMO  Total  Effort  (effort  on  x  axis) 

2. 

CORADMO  Total  Effort  (years  on  x  axis) 

3. 

CORADMO  Total  Effort  (only  for  CD,  K  and  EK) 

4. 

CORADMO  Development  (E+C)  Effort  with  COCOMO II  Development  (E+C) 
Effort 

5. 

CORADMO  Development  (E+C)  Effort  with  COCOMO  II  Development  (E+C) 
Effort  (only  for  CD,  K  and  EK) 

6. 

CORADMO  Total  Schedule  (schedule  on  X  axis) 

7. 

CORADMO  Development  (E+C)  Schedule  with  COCOMO  II  Development 
(E+C)  Schedule(only  for  CD,  K  and  EK) 

8. 

CORADMO  Development  (E+C)  Schedule  with  COCOMO  II  Development 
(E+C)  Schedule 

9. 

New/Current  CORADMO  Total  (I+E+C)  Effort  with  Default  CORADMO  Total 
(I+E+C)  Effort 

10. 

New/Current  CORADMO  Total  (I+E+C)  Schedule  with  Default  CORADMO 
Total  (I+E+C)  Schedule 

Table  2.  Final  Result  Charts 
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F.6  Sensitivity  Analysis  Example 


One  of  the  reasons  for  allowing  input  of  new  values  for  the  drivers  is  to  permit  sensitivity 
analysis.  Suppose  it  is  believed  that  the  impact  of  reuse  and  very  high  level  languages  has  been 
over  estimated,  and  that  only  50%  of  the  originally  estimated  impact  seems  to  be  justified. 

F.6.1  COCOMO-II.  1998  Driver  Modification 


First,  the  impacted  COCOMO-II.  1998  drivers  need  to  be  adjusted.  Since  there  is  no 
explicit  driver  for  reuse,  but  rather  the  effective  size  is  modified  to  reflect  the  decreased  amount 
of  reuse  and/or  use  of  a  very  high  level  language.  This  can  be  accomplished  by  filling  each  cell 
in  the  “new”  row  except  the  baseline,  which  remains  the  same,  by  a  formula  that  subtracts  fifty 
percent  of  the  difference  between  the  baseline  and  the  default.  The  new  values  are  shown  in 
Figure  15. 
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Size  (KSLOC)  is  the  primary  determinant  of  software  effort  in  COCOMO  II  (and  other  SCE  models). 
For  COCOMO  II,  effective  size  =  f(KSLOC  or  FP.  BRAK,  ADSI,  DM,  CM,  IM,  SU,  UNFM). 

j®  ®  100  KSLOC  embedded,  high-assurance,  real-time  (EHART)  software  application.  ! 

ALL.  Only  fifty  percent  impact  on  size  due  to  reuse/RVHI 


Figure  15.  Fifty  Percent  SIZE  Impact 


The  “new”  values,  except  baseline,  in  the  table  above  the  chart,  like  that  in  E632,  were 
calculated  with  a  formula  like:  “=$D659-($D659-E659)*(50/100)”.  The  new  values  are  reflected 
in  the  chart  below.  Also,  the  rationale  is  noted  in  the  section  following  the  chart. 

F.6.2  CORADMO  Driver  Modification 

Since  RVHL  reflects  the  impacts  of  reuse  and/or  very  high  level  languages,  it  is  the  only 
driver  that  needs  to  be  adjusted.  Figure  16  shows  the  new  values  for  the  RVHL  schedule 
multiplier  for  the  inception  stage. 
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ALL:  Only  50%  impact  as  part  of  sensrtivity  analysis 

Figure  16.  RVHL  Inception  Stage  Adjustment  for  50%  Impact 


The  “new”  values,  except  baseline,  in  the  table  above  the  chart,  like  that  in  E4,  were 
calculated  with  a  formula  like:  “-SDI-fSOI-EIjnSO/lOOr.  The  new  values  are  reflected  in  the 
chart  below.  Also,  the  rationale  is  noted  in  the  section  following  the  chart. 

Similarly,  Figure  17  shows  the  new  values  for  the  RVHL  schedule  multiplier  for  the 
elaboration  stage  as  well  as  the  rationale  for  the  modification. 


F.6.3  Sensitivity  Analysis  Results:  Technology  Impact  Estimates 

The  effort  and  schedule  results  after  applying  the  COCOMO-II.1998  and  CORADMO 
driver  modifications  identified  above  are  shown  in  Figure  18.  This  result  can  be  compared  tome 
default  driver  settings  results  shown  in  Figure  20.  CORADMO  Total  Effort  in  Figure  18  refers 
to  the  same  values  as  CORADMO  Total  (I+E+C)  Effort  in  Figure  20. 
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Figure  19  is  a  comparison  of  COCOMO-II.1998  only  results  and  the  final  CORADMO 
with  only  50%  of  the  reuse  and  high  level  language  impact. 
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Figure  19.  One  of  the  comparisons  of  COCOMO-II.1998  only  results  and  Final  Results 

Here,  both  the  COCOMO-II.1998  set  of  calculations  and  the  final  results  calculations  are 
shown  in  the  table  above  the  chart.  Again,  the  final  results  row’s  values  will  contain  the  results 
based  on  the  “currenf  ’  CORADMO  driver  values,  and  thus  may  have  changes  anytime  there  is 
input  in  the  “new”  row  of  the  drivers.  While  only  the  data  associated  with  the  top  row  of  the 
table,  which  contains  the  COCOMO-II.1998  calculation  results,  is  shown  m  the  chart,  the  final 
results  values  are  evident  due  to  the  dashed  lines  appearing  in  the  chart. 

For  sensitivity  analyses,  the  set  of  comparison  charts  is  especially  useful.  They  show  the 
overall  effort  and  schedule  results  using  the  default  driver  values  and  the  new/current  driw 
values.  Along  with  each  chart  are  copies  of  the  rows  of  the  appropriate  data  from  "CORADMO 
Data"  sheet.  Figure  20  shows  a  comparison  of  final  CORADMO  results  for  default  and  new 
drivers  (with  the  only  driver  change  being  SIZE  (change  amount  reduced  by  50%)). 
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Figure  20.  Comparisons  of  Effort  Final  Results  for  Default  and  New  Drivers 

Here,  both  the  default  and  new  final  results  calculations  are  shown  in  the  table  above  the 
chart.  Again,  the  new  final  results  row’s  values  will  contain  the  results  based  on  the  “current” 
CORADMO  driver  values,  and  thus  may  have  changes  anytime  there  is  input  in  the  “new”  row 
of  the  drivers.  While  only  the  data  associated  with  the  bottom  row  of  the  table,  which  contains 
the  new  calculation  results,  is  shown  in  the  chart,  the  default  results  values  are  evident  due  to  the 
solid  lines  appearing  in  the  chart. 

The  corresponding  schedule  final  results  comparison  is  shown  in  Figure  21. 
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G.  Technical  Transition  Activities 

The  Bayesian  approach  for  calibrating  parametric  models  to  a  mix  of  expert-consensus- 
determined  values  and  project  data  [Chulani  et  al.,  1998]  has  been  applied  to  COCOMO  11.1998, 
and  results  reported  at  the  1997  COCOMO/Software  Cost  Modeling  Forum,  the  1998 
International  Conference  on  Software  Engineering,  and  the  1998  ISPA/SCEA  Conference.  It  is 
also  being  applied  to  USC-CSE’s  Constructive  COTS  Integration  cost  model  (COCOTS)  and 
Constructive  Quality  Estimation  Model  (COQUALMO),  initially  reported  at  the  1997  California 
Software  Symposium  [Chulani,  1997]. 

These  and  the  other  recently  completed  models  discussed  in  this  report  (CORADMO, 
COSSEMO,  and  the  Software  Technology  Impact  Assessment  Model)  were  presented  at  the 
1998  COCOMO/Software  Cost  Modeling  Forum  in  October  1998.  Concurrently  with  this 
Forum,  USC-CSE  held  an  Affiliates’  Workshop  on  COCOMO  II  Extensions  to  provide  the 
models  to  the  Affiliates  and  discuss  their  usage  and  refinement. 

The  preliminary  results  of  the  KBS  A  Life  Cycle  Evaluation  were  briefed  to  AFRL/IFTD 
personnel  at  Rome  on  July  8,  1998.  The  final  results  will  be  briefed  to  selected  Air  Force  and 
DoD  software  technology  managers. 

As  indicated  in  Section  I,  we  are  also  making  the  results  and  model  available  via  the 
USC-CSE  Web  site. 
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H.  Summary  and  Lessons  Learned 

1.  The  KBSA  Advanced  Development  Model  was  not  sufficiently  robust  and  scalable  to 
provide  appreciable  benefits  to  critical  DoD  software  projects.  We  found  its  framework 
difficult  to  scale  to  large  projects,  and  its  toolset  to  be  small  with  respect  to  large  project 
needs. 

2.  However,  the  KBSA  concepts,  if  otherwise  realized,  have  strong  potential  for 
reducing  costs  and  schedules  for  DoD-critical  warfighting  software  applications.  The 
key  to  realizing  this  potential  is  to  develop  the  KBSA  technology  in  the  context  of  DoD 
warfighting  domain  knowledge.  For  KBSA  Applications  Generation,  this  implies 
focusing  on  DoD  warfighting  areas  where  a  good  start  exists  towards  domain 
engineering,  domain  architecting,  and  development  and  use  of  reusable  domain  assets. 
Some  of  the  USAF/ESC  Product  Line  areas  are  good  candidates,  as  are  some  avionics, 
missile,  and  sensor  processing  domains. 

For  KBSA  Decision  Support,  this  involves  both  domain-specific  decision  aids  and  more 
generic  decision  aids  for  security,  survivability,  risk  assessment,  architecture  definition, 
and  process  definition  and  management. 

3.  The  KBSA  Decision  Support  area  provides  an  attractive  new  direction  for  Air  Force 
and  DoD  software  technology  investments.  Knowledge-based  and  agent-based  mixed- 
initiative  technical  approaches  are  generating  high  payoffs  in  decision  support  areas 
similar  to  software  engineering  such  as  manufacturing  and  logistics.  A  similar  initiative 
for  software  engineering  would  build  on  existing  technology  in  attractive  new  directions. 

4.  EDCS  technology  investments  and  payoffs  have  both  commonalities  and 
complementarities  with  respect  to  KBSA  technology  investments  and  payoffs.  To  some 
extent,  good  portions  of  a  KBSA  Applications  Generation  program  have  been  pursued 
under  EDCS.  A  KBSA  Decision  Support  program  could  build  upon  and  provide  new 
directions  for  some  current  EDCS  efforts  in  such  areas  as  rationale  capture,  design  webs, 
and  architecture  technology. 

5.  The  Software  Technology  Impact  Analysis  spreadsheet  model  provides  a  useful  tool 
for  assessing  alternative  software  technology  assessment  strategies.  It  enables 
technology  managers  to  relate  candidate  technologies’  effects  on  software  cost  and 
schedule  drivers  to  resulting  project  cost  and  schedule  improvements.  It  is  well 
calibrated  and  baselined  with  respect  to  current  software  practice  and  commercial 
technology  and  productivity  trends. 

6.  There  is  no  software  technology  silver  bullet  for  DoD  warfighting  systems,  including 
reliance  on  commercial  technology.  A  mixed  strategy  of  commercial  technology  and 
complementary  DoD  warfighting  software  technology  is  necessary  to  address  DoD’s  full 
range  of  needs. 
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/.  World  Wide  Web  Home  Pages  and  References 

I.l  World  Wide  Web  Home  Pages 

1 .  use  Center  for  Software  Engineering: 
http  .’//sunset. use  .edu/ 

2.  Technical  Reports: 

http://sunset.usc.edu/TechReports/electronicopy.html 

3.  COCOMO II  Research  Program  and  Results: 
http://sunset.usc.edu/COCOMOII/cocomo.html 

4.  Software  Technology  Impact  Assessment  Model: 
http://sunset.usc.edu/COPROMO/KBSA_LCE/kbsa_lce.html 
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